Auto accidents cost each American more than $1,000 a year, 2½ times the cost of the traffic jams that frustrate the nation's drivers, according to a report issued by the American Automobile Association (AAA). According to the AAA report, accidents cost $164.2 billion each year, which based on the methodology used in the report, comes to an annual per person cost of $1,051. AAA said the study that quantified the cost of traffic accidents was conducted by Cambridge Systematics and considered costs from medical care, emergency and police services, property damage, lost productivity and quality of life. “Nearly 43,000 people die on the nation's roadways each year,” said AAA President and CEO Robert L. Darbelnet in a report. “Yet, the annual tally of motor vehicle-related fatalities barely registers as a blip in most people's minds.”
Vehicle traffic continues to grow at a rate that far outpaces the supply of new roads and highways. For example, a study on California roadways, Beyond Gridlock: Meeting California's Transportation Needs in the Twenty First Century Surface Transportation Policy Project, May 2000, incorporated herein by reference, provides statistics showing that demand is far outpacing supply. Over a 13-year period, the number of vehicle miles traveled increased by 45% while new road facilities increased by 5% to 26% depending on the road type. Therefore, traffic demand increased at approximately twice the rate of new facilities over that period.
Automobile transportation may also benefit from new technologies that allow for more cars to use existing roads and facilities more efficiently. For example, these technologies may include smart traffic lights, vehicle transponders and other on-board systems. Cooperative technologies receive a lot of attention for potential future vehicle applications. Pioneered in commercial aviation, use of on-board cooperative devices, such as transponders, allows for the communication of intent between users and third parties. In aviation, all aircraft are required by law to carry transponder devices in regulated airspace for applications such as surveillance and collision avoidance. With something like 10,000 commercial and 250,000 general aviation aircraft in existence today, the use of transponders is regulated, mainly because aviation is inherently global and governments have heretofore been responsible for air traffic control.
For example, a vehicle transponder for pre-emption of traffic lights, is presented in a NASA Tech Brief, dated September 2006, and incorporated herein by reference. That tech brief describes when the unit at an intersection determines that a vehicle is approaching and has priority to preempt the intersection; it transmits a signal declaring the priority and the preemption to all participating vehicles in the vicinity. If the unit at the intersection has determined that other participating vehicles are also approaching the intersection, then the unit also transmits, to the vehicle that has priority, a message that the other vehicles are approaching the same intersection. The texts of these messages, plus graphical symbols that show the directions and numbers of approaching vehicles are presented on the display panel of a computer that is part of the transponder.
While these systems have been designed, built and no doubt work effectively, the problem with full-scale implementation is institutional and not technical. The big issue with cooperative devices is that all vehicles need to be equipped to provide benefits overall. As in aviation, if one vehicle in a particular scenario is not equipped, the entire system is rendered useless and may be unsafe. Thus a need exists in the art for a system, which is not cooperative in order to maximize benefits and operate in a mixed equipage scenario.
The U.S. DOT reported on the benefits of smarter traffic light management in a 2006 report, presented at www.benefitcost.its.dot.gov/ITS/benecost.nsf/ByLink/BOTM-October2006, incorporated herein by reference. In the Tysons Corner area of Northern Virginia, approximately 40 signalized intersections were connected to a temporary operations center. In the control room, operators monitored traffic conditions and retimed signals as necessary to improve traffic conditions. The DOT analysis estimated the system saved motorists approximately 20 million dollars annually. Stops were reduced by approximately 6 percent (saving 418 thousand dollars), system delays decreased by an estimated 22 percent (18 million dollars), and fuel consumption improved by an estimated 9 percent (1.5 million dollars). Total annual emissions of CO, NOx, and VOC were decreased by an estimated 134.6 thousand kilograms.
Other new technologies proposed for vehicle traffic management include the use of so-called intelligent beacons. U.S. Pat. No. 6,714,127, entitled Emergency Warning Intelligent Beacon System for Vehicles, incorporated herein by reference, describes a beacon system located at various points of interest to transmit local information to nearby motorists. Potential uses of the system include a speed limit beacon installed on a speed limit sign to reflect current or recommended speed limit based on weather conditions, ice, rain, potential hazards, and the like. Another use is as a fog zone beacon installed in known fog zone areas where motorists are alerted of fog zone conditions ahead. Other uses of beacons include announcing freezing bridge surfaces, frozen road surface conditions, railroad crossings, and the presence of hazardous materials.
Use of radar sensors for various vehicle applications is well described in the prior art. Radar sensors are usually used to assist parking, monitor blind spots, anticipate collisions, starting and stopping operation or during driving with distance monitoring, and to regulate separation through cruise control operation. U.S. Pat. No. 7,243,013, entitled Vehicle Radar-Based Side Impact Assessment Method, incorporated herein by reference, describes the use of radar sensors using a single radar sensor mounted on each side of the vehicle to generate a range and range-rate value for detected target objects, and a controller coupled to each radar sensor. The controller calculates estimated target object speed, angle of the target object line of travel, and a shortest distance value from the sensor to the target object line of travel, and compares the shortest distance value and a change in the angle value to respective threshold values for potential collision threat assessment.
U.S. Pat. No. 7,268,732, entitled Radar Sensor For Use With Automobiles, incorporated herein by reference, describes the use of a different frequency band and modulation technique to monitor the near field region around a vehicle. This patent also states that current radar sensors are normally used for remote object detection, and that, for near field observations, high spatial resolution is important for separation as well as angle, whereas the angular information is less important for large separations. For monitoring of separation at large range, radar sensors are conventionally used having a frequency of approximately 76 Gigahertz. These frequencies have some disadvantages, however, and frequencies of approximately 24 Gigahertz are better for near-field monitoring.
On-line magazine CNET offers reviews of various new consumer electronics items, including one by Bonnie Cha, of the Garmin Nuvi series of car GPS units, published on Nov. 20, 2006, incorporated herein by reference. The reviewer notes many newer features are now being integrated with GPS devices such as Bluetooth, so it may be used hands-free to make and accept phone calls. If a number is listed for a point of interest, the Nuvi 660 model may dial out to that business with a press of a button and traditional voice-guided directions are automatically muted during incoming calls. There are also options to send text messages, synchronize cellular phone address books and call log, and dial by voice Like most of the units on the market the maps are available in 2D and 3D view with day and night colors, and the view may be changed so that either north or the direction of travel or always at the top of the screen. Plus and minus icons on the map screen allow you to zoom in and out, and there's also a trip information page that displays car speed, direction, trip time, and so forth. The Nuvi 660 has a database with all the major categories and more specific ones; one may search for restaurants by type of cuisine, for example. While, as for the mobile phone industry, features are constantly added to in-car GPS units, these features are mainly limited to the somewhat obvious addition of user applications that run on the GPS unit's processor, with a lesser degree of integration to the GPS unit's main routing and guidance functions.
New technologies envisioned for vehicles also include the use of signaling. In the weblog blog.mboffin.com/postaspx?id=2208, on June 2007, incorporated herein by reference, the participants in the forum discuss the idea of using various lights to show the driver's use of controls. For example, the question is posed that “you have brake lights to know when someone has their foot on the brake pedal, so why not acceleration lights to know when they are pushing on the accelerator pedal?” In this example, the posters go on to discuss variable headlight intensity related to the car's acceleration, based on acceleration pedal movement. However, they quickly point out all of the impracticalities of such a scheme due to variations in different car headlamp intensities, not to mention differing ambient light conditions.
In recent years, some signaling lights have been added to cars including the third Center High Mount Stop Lamp (CHMSL), and the use of indicator lights on car mirrors and side panels. According to en.wikipedia.org/wiki/Automotive_lighting, incorporated herein by reference, in 1986, the United States National Highway Traffic Safety Administration and Transport Canada mandated that all new passenger cars have a Center High Mount Stop Lamp (CHMSL) installed. Referred to as the center brake light, or the “Dole light,” after the then. Secretary of Transportation, Elizabeth Dole, this light provides a deceleration warning to following drivers, whose view of the braking vehicle's regular stop lights is blocked by interceding vehicles. It also helps to distinguish brake signals from turn signals in North America, where red rear turn signals identical in appearance to brake lights are permitted. According to NHTSA Technical Report Number DOT HS 808 696: The Long-Term Effectiveness of Center High Mounted Stop Lamps in Passenger Cars and Light Trucks, by Kahane, Charles J. and Hertz, Ellen (1998), incorporated herein by reference, the CHMSL is credited with reducing collisions overall by about 5%.
Bavarian Motor Werks, of Germany, has implemented a technology known as “adaptive brake lights” where the intensity or number of brake lights illuminated is altered depending upon the type of braking. In a normal braking situation, standard brake lights meeting DOT or other requirements are activated. However, in a panic stop (as measured by pedal pressure or accelerometers) additional brakes lights are illuminated and/or existing brake lights are illuminated at a higher intensity to better catch the attention of a following driver.
Taking the use of onboard systems and the smart car concept to a logical conclusion, there is talk of cars that drive themselves. In an interview with the British Broadcasting Corporation (BBC) on Nov. 5, 2007, published on BBC.co.uk and incorporated herein by reference, Larry Burns, GM's vice-president for research and development and strategic planning, stated that self-driving cars might be on the road by the year 2015. That article also included a description of a competition held for eleven driverless cars that had to navigate around a 60 mile course without operator intervention. The cars had various sensor devices onboard including radar and Lidar (light detection and ranging), GPS navigation, and databases.
Published U.S. Patent Application 2007/0276581, entitled Alerting a Vehicle Operator to Traffic Movement, incorporated herein by reference, identifies a zone around a host vehicle and identifies a target vehicle in the zone. The speed and location of the target vehicle are monitored and an alert is generated in the host vehicle if the target vehicle is moving outside of the zone at a speed higher than a minimum speed and the host vehicle is stationary. The system is used when a vehicle that is traveling in a series of consecutive vehicles stops due to traffic lights or a traffic jam, and the operator often fails to move the vehicle forward immediately after the traffic light changes or the traffic jam is cleared. This failure to move the vehicle forward may cause further delays or traffic jams to occur. The technique relies on forward-looking radar or other sensors to detect the motion of the vehicle in front.
While Published Patent Application 2007/0276581 addresses one of the issues relating to increasing efficiency on the nation's roads, there are many more areas where traffic throughput and latency may be improved. According to U.S. DOT statistics, listed in the 2006 Transportation Statistics Annual Report, Research and Innovative Technology Administration, Bureau of Transportation Statistics, and incorporated herein by reference, the number of cars owned per U.S. household has increased by over 60% between 1969 and 2001, as illustrated in FIG. 1. This is but one of many indicators of the ever-increasing growth of vehicle traffic in the United States, and elsewhere.
According to the 2007 Urban Mobility Report, by D. Schrank and T. Lomax, of the Texas Transportation Institute, Texas A&M University System, incorporated herein by reference, traffic signal timing may be a significant source of delay. The report states that much of this delay is the result of managing the flow of intersecting traffic, but some of the delay may be reduced if the traffic arrives at the intersection when the signal is green instead of red. FIG. 2 illustrates the potential impact of traffic light coordination in the context of other possible operations treatments. The authors go to summarize at a high level, that each peak time traveler in an urban area is subjected to almost 40 hours of delay annually (see FIG. 3). The authors further conclude that non-peak travelers are subjected to approximately 30 hours of delay annually. In this context, delay is defined as the extra time spent traveling due to congestion. Travel delay calculations were performed in two steps—recurring (or usual) delay and incident delay (due to crashes, vehicle breakdowns, etc.). Recurring delay estimates were developed using a process designed to identify peak period congestion due to traffic volume and capacity. Delay caused by other events is not included in the recurring delay estimate. Generally, these events may be categorized as one of the seven sources of unreliability, including Traffic Incidents, Work Zones, Weather, Fluctuation in Demand, Special Events, Traffic Control Devices, and Inadequate Base Capacity.
There are systems described in the Prior Art that provide warnings or other situational awareness information to the driver of a vehicle. U.S. Pat. No. 7,274,287, incorporated herein by reference, describes a warning and information system for a motor vehicle, which outputs information that is below a conscious threshold of perception. At least one signal source located in the peripheral field of vision of the user is provided, and its output signals are variable by adjusting their color, intensity, frequency, timber, or loudness. The patent describes that modern warning and information systems should warn the vehicle driver of hazardous situations, which previously had to be recognized by the driver alone. For example, some complex ambient detection systems based on radar, infrared or image processing technologies have been implemented in Mercedes Benz S-class motor vehicles, where these technologies support the longitudinal and transverse guidance of the vehicle through visual or acoustic information.
These types of information systems are usually designed so that a hazard warning is provided only when the driver has not personally perceived the hazard, otherwise the warning information has no utility. German patent document DE 199 52 506 C1, incorporated herein by reference, describes a system which displays information in the form of images or symbols at least once for a brief period of time in the primary field of vision of the operator, the period of time being below a conscious threshold of perception by the user and above an unconscious threshold of perception. The purpose being to provide a warning and information system for a vehicle, which may increase traffic reliability and support the driver of the vehicle with longitudinal and transverse guidance tasks.
There are systems described in the prior art that use image processing to aid in situational awareness. U.S. Pat. No. 7,230,538, entitled Apparatus and Method for Identifying Surrounding Environment by Means of Image Processing and for Outputting the Results, incorporated herein by reference, describes an electronic apparatus identifying the surrounding environment by means of image processing and outputting the results for use by blind people. That patent describes a guide for sight-limited pedestrians or unmanned vehicles at traffic lights and crosswalks and includes the ability to detect the presence of traffic and crosswalk lighting and changes in the lights, annunciated through an audio output, such as an earphone. The processing device applies hue analysis and geometric analysis to identify traffic signals and markings. A flowchart showing the operation of this invention is provided in FIG. 4 of the present application.
Other inventions using light detection include Taiwan patent publication No. 518965, entitled Speech Guide Glasses, incorporated herein by reference, which describes glasses comprising a sensor and a speech earphone. The sensor has two functions, the first being to sense the color of a traffic light in front of the user, and the second being the detection of obstacles ahead by receiving a reflected IR beam sent out by the sensor. The invention uses an RGB filter to process the received light prior to recognizing a traffic light. However, the effectiveness of the system is not known as all visible light may be decomposed into RGB primary colors and the data flow for the recognition process may be very substantial. Furthermore, the effectiveness of this approach in various environments and lighting conditions is unknown.
Published U.S. Patent Application 2008/0013789, entitled Apparatus and System for Recognizing Environment Surrounding Vehicle, incorporated herein by reference, describes a two camera system used to detect and recognize the environment surrounding a vehicle, as illustrated in FIG. 5. The apparatus of FIG. 5 is mainly aimed at recognizing various road markings. Conventional systems use a rear-facing camera to recognize objects surrounding the vehicle. The images are road surface markings at the lower end of a screen, making it difficult to predict specific positions of road surface markings. Further, the angle of depression of the camera is large, and has a short period of time to acquire the object leading to low quality recognition and false marking recognition. Results of recognition including the object type, position, angle, and recognition time from an additional forward-looking camera are used to predict specific timing and position of field of view of the rear facing camera, at which the object appears. Recognition logic parameters of the rear facing camera and processing timing are then optimally adjusted. Further, luminance information of the image from the forward-looking camera is used to predict possible changes to be made in luminance of the field of view of the rear-facing camera. Gain and exposure time of the rear-facing camera may then be adjusted accordingly.
Rear-view cameras for vehicles have been known in the art. Such cameras have been proposed as replacements for side rear view mirrors, to reduce air drag, or as replacements for interior rear view minors (e.g., backup camera) or the like. For trucks and motor homes, such cameras and displays have been available for years, and are also offered as aftermarket add-on items. Many manufacturers are offering such backup cameras as options in European models and in some U.S. models. However, these devices merely serve as cameras and video displays, which supplant or augment traditional rear-view mirror displays. They do not provide any automated detection features.
Volvo recently introduced a system known as BUIS, or Blind Spot Information System, which now recognizes cars and motorcycles with a camera-based monitoring system that keeps a watchful eye on the ‘blind’ area alongside and offset rear of the car. When another vehicle (motorcycle, car or truck) enters this zone—an area of 9.5 meters by 3.0 meters—a yellow warning light comes on beside the appropriate door minor in the driver's peripheral view. The driver is thus given an indication that there is a vehicle very close alongside. This visual information gives the driver added scope for making the right decisions in such driving situations. A digital camera is installed on each door minor. This small camera captures 25 images per second, and by comparing each frame taken, the system is able to recognize that a vehicle is within the BUS zone. The system's software is programmed to identify cars as well as motorcycles, in daylight and at night. Since BUS is camera-based, it has the same limitations as the human eye does. This means the system may not function in conditions of poor visibility, for instance in fog or flying snow. If that happens, the driver receives a message that BUS is not in action. BUS is configured not to react to parked cars, road barriers, lampposts and other static objects. The system is active at all speeds above 10 km/h. It reacts to vehicles that are driven a maximum of 20 km/h slower and a maximum of 70 km/h faster than the car itself.
Bavarian Motor Werks has implemented a similar system in the 2009 7-series sedan, utilizing radar sensors to alert motorists if a vehicle is in their blind spot during lane changes. Building on the impressive list of innovations, the 2009 7 Series is the first BMW to feature Lane Change Warning combined with Lane Departure Warning. The latter is a system that first appeared on BMW 5 and 6 Series models and uses a camera to monitor road markings. Should the driver start to stray out of lane, a gentle vibration of the steering wheel provides an alert. However, courtesy of two radars located at either side of the rear bumper, Lane Change Warning adds another level of driver safety. The sensors constantly scan the blind spot either side of the vehicle, up to a distance of 60 meters, and alert the driver to the presence of another vehicle with a triangular symbol in the door minor housing.
Although there is generally a lot of talk about the introduction of new traffic management infrastructures and data linking between vehicles, it may take many years for any type of system that is cooperative and relies on high vehicle equipage scenarios. According to U.S. DOT statistics, listed in the 2006 Transportation Statistics Annual Report, Research and Innovative Technology Administration, Bureau of Transportation Statistics, shown in FIG. 6, and incorporated herein by reference, the average age of an in-use passenger car in the United States has grown to over 9 years in the past decade. This means that any equipage scenario should consider this length of ownership, i.e., maybe up to 20 years or so for many vehicles to be replaced. Hence, there is a need for solutions that are autonomous and not cooperative in order to gain benefits in the foreseeable future.
In their book, Introduction to Remotely Sensed Data, incorporated herein by reference, Harrison and Jupp describe the human sensation of color due to the sensitivities of three types of neurochemical sensors (which are present in the cones of the retina) to different wavelengths in the visible region of the electromagnetic spectrum. They describe each sensor association with one type of cone and responding to a range of wavelengths, with varying sensitivity. One type of sensor is maximally sensitive to short wavelengths with a peak response at approximately 0.44 μm. This is often referred to as the blue sensor and is insensitive to wavelengths longer than 0.52 μm. The second sensor has peak sensitivity at 0.53 μm, or green light. The third is referred to as the red sensor although peak sensitivity actually occurs at 0.57 μm, which is the wavelength of yellow light. However; of the three, the third sensor has the highest absorption of red light.
Many camera systems emulate the functions associated with the human sensation of color. Color filter arrays are used to arrange color filters on photo sensors, such as the Bayer filter mosaic, described in U.S. Pat. No. 3,971,065, incorporated herein by reference, which refers to an arrangement of color filters used in many digital cameras image sensors to create a color image. The filter pattern is 50% green, 25% blue and 25% red. The Bayer filter is common on consumer digital cameras, and alternatives include the CYGM filter (cyan, yellow, green, magenta) and RGBE filter (red, green, blue, emerald), and the Foveon X3 sensor, which layers red, green, and blue sensors vertically rather than using a mosaic; or using three separate CCDs, or one for each color. There are various other types of sensors that filter based on color such as described in Published U.S. Patent Application No. 2007/0024879, entitled “Processing Color and Panchromatic Pixels,” and incorporated herein by reference, and Published U.S. Patent Application No. 2007/0145273, entitled “High-Sensitivity Infrared Color Camera,” also incorporated herein by reference.
There are various systems available on the market including software and hardware for data analysis, pattern recognition and image processing, however; the task of identifying changes in status of traffic lights is relatively simple in terms of target association. As shown in FIG. 7, the progression of a light change from red (stop) 20, 25 to green (go) 30, 35 is generally vertical, with the red light geometrically directly above the green light. As is known in the art, red (stop) light 20,25 may include a strobe light and one or more of green (go) light 30,35 may incorporate a green arrow. The range of the scale (i.e., the proximity of the red to the green) is also quantifiable based on typical distances from the intersection to the position of the light. There are other identifiable patterns, such as the horizontal relationship between dual lights, as shown in FIG. 7 where the dual reds 20, 25, and the dual greens 30, 35, are shown.
Gadberry, U.S. Pat. No. 6,108,141, issued Aug. 22, 2000 and incorporated herein by reference discloses a Fresnel lens type device that may be attached to a windshield, so that a driver can more easily view a traffic signal. These types of Fresnel lenses are used in the Recreational Vehicle (RV) and bus industries and for vans and other large vehicles where view from the rear window is limited. Gadberry proposes a smaller version of these stick-on lenses (which use static cling to attach to a window) so a driver can more easily see a traffic light. He proposes, in one embodiment, giving them away free as an advertising promotion.
Schofield et al., U.S. Pat. No. 7,388,182, issued Jun. 17, 2008, and incorporated herein by reference, discloses an image sensing system for controlling an accessory or headlight of a vehicle. One embodiment appears to be a variation on the automatic headlight dimming devices known in the art since the Cadillac “twilight sentinel” of the 1950's. However, Schofield discloses being able to detect oncoming headlights and also brake lights of cars ahead. In one alternative embodiment (Col. 12, lines 22-29) he mentions being able to detect the spectrum of a traffic light, so as to determine whether a light has turned yellow or red from green.
Schofield would appear to suffer from a number of technical problems. While it is possible to detect the frequency of light from a traffic signal, other lights (advertising, automotive, and the like) may also be at the same frequency, possibly generating false signals. In addition in complex or sequential intersections, where multiple traffic lights are within the field of view of the device, it may be difficult to discriminate the correct signal for a given intersection and lane within an intersection. A green arrow, for example, may generate a signal on a green wavelength, but it does not mean that a car in the center lane may proceed.
Doan, U.S. Pat. No. 6,985,073, issued Jan. 10, 2006, and incorporated herein by reference, discloses an apparatus for monitoring traffic signals and alerting drivers. Doan discloses a mechanism for monitoring status of traffic signals. A photodiode sensor mounted on the rear view minor (FIG. 2) senses traffic stoplights and taillights of other vehicles. Doan uses a wireless system to send data to off-vehicle receivers for accident reconstruction and law enforcement capabilities.
Doan is relevant to the present discussion in that he discloses monitoring traffic signals and alerting a motorist. However, his photodiode system would seem to have the same issues as Schofield, as it would not detect a traffic signal unless the diode was aimed properly. In addition, it is not clear how the system would react to multiple signals in the same area (green arrows, adjacent or subsequent lights, complex intersections) so the incidence of false alarms could be common (with dangerous results, if a driver relies upon a false green indicator). It also appears that Doan's device might trigger on other types of lighting (signage, advertisements, neon, car lights, etc.) as a photodiode is not very discriminating. Areas with large numbers of colorful lighted signs (e.g., Times Square) might overwhelm such a simple detector. While the Doan patent discusses wireless communications, it does so only in terms of broadcasting from the vehicle to a non-vehicle, for forensic and law enforcement purposes.
Kubota, U.S. Pat. No. 7,398,076, issued Jul. 8, 2008, and incorporated herein by reference, discloses a method for producing traffic signal information. This patent uses a machine vision system to determine whether an image of a traffic signal is present in the field of view. This patent represents an improvement over Doan, as the photodiode of Doan is probably too primitive to reliably detect traffic signals. Image sensing has its own problems, of course. Again, multiple traffic lights in the field of view may confuse the system. Moreover, even the image of a traffic light (e.g., on a billboard) may be confused with an actual signal.
Bae, Published U.S. Patent Application No. 2007/0276581, published Nov. 29, 2007, and incorporated herein by reference, discloses a method for alerting a vehicle operator to traffic movement. The system uses vision cameras and other sensors to detect if a vehicle ahead of a stopped car has moved and alert the driver to accelerate accordingly. This invention appears to be narrowly focused on one particular driver scenario.
The Prior Art references cited above demonstrate a long felt need in the art for automated and semi-automated vehicle system to improve driver situational awareness system for use in intersections. As automobile use expands worldwide, traffic congestion increases, driver distractions increase, and driver skills continue to deteriorate, the number of accidents at intersections will increase over time, resulting in death, personal injuries, and increased costs for motorists for insurance and repairs. Systems are needed to improve driver situational awareness as more and more unskilled drivers take to the road and as highway congestion increases.